fix: honor submodel device placement across inference calls by damnwenxi · Pull Request #3706 · modelscope/FunASR

damnwenxi · 2026-09-14T01:22:43Z

Summary

An explicit vad_kwargs={"device": "cpu"} was overwritten when ASR used an accelerator. Shared inference options could overwrite the device again, and the speaker model received ASR configuration instead of its own.

Honor the existing nested device options for VAD, punctuation and speaker models, inheriting the resolved ASR device only when omitted. Copy caller-owned submodel dictionaries before filling defaults. Preserve construction-time placement through runtime merges and repeated calls, and pass the speaker model its own resolved configuration. Punctuation is covered both with and without VAD.

Related to #3701 and the interface proposed in the maintainer response.

Type of change

Bug fix
Documentation

Validation

Using the existing local Python environment against this checkout:

python -m unittest discover -s tests -p test_submodel_device.py -v
# 4 passed
python -m unittest discover -s tests -p test_punc_model_none.py -v
# 16 passed
python -m unittest discover -s tests -p test_python_api_docs_contract.py -v
# 10 passed
python -m compileall -q funasr examples tests
# passed; existing invalid-escape SyntaxWarnings in format5res.py
git diff --check
# passed

A follow-up real Apple Silicon validation was also posted in the PR discussion: Apple M4/MPS, 71m36s local audio, offline weights, encoder hooks checking feature/weight devices, warmup plus two full runs per placement mode. The best mixed placement was ASR=MPS, VAD=CPU, punctuation=MPS.

The new regressions exercise actual AutoModel construction, generate/inference orchestration and per-model inference boundaries with registry stand-ins. They cover explicit overrides, omitted-device inheritance, caller-dictionary isolation, shared non-device runtime options, two successive calls, speaker-only configuration, both punctuation routes, and unavailable-device fallback.

All stand-in weights remain on CPU. Mocked CUDA availability is used only for configuration labels, not as evidence of accelerator execution. CPU feature/weight agreement is checked in the stand-ins. No pretrained weights are downloaded.

User impact

Allows long-audio users to keep VAD on CPU while ASR uses MPS/CUDA through public configuration, removing the need to replace internal VAD objects or patch ComputeScores. English and Chinese Python API docs include the same mixed-device example.

The original Apple M2 timings in #3701 (approximately 259s all-MPS versus 156s mixed placement for 121-minute audio) motivate this change; they were measured with the earlier workaround, not this patch.

Notes for reviewers

Real mixed-device validation has been added in PR comments. On Apple M4/MPS, a local 71m36s audio file was run with the PR checkout, offline model loading, fresh subprocesses per mode, a 10s warmup, and two full-length runs per mode. Read-only encoder hooks verified feature/weight device agreement for ASR, VAD and punctuation. mixed_vad_cpu (ASR=MPS, VAD=CPU, punc=MPS) was fastest on that workload: 84.38s and 123.43s versus all-MPS at 161.87s and 205.87s; all modes produced matching text hashes, timestamp counts and sentence counts. This is an Apple M4 local-audio validation, not the original 121-minute M2 workload.

Runtime device overrides are ignored for already-loaded ASR and submodels; other runtime options still merge normally. This makes the existing documented no-runtime-migration contract explicit. Export retains its separate behavior of copying and moving the model.

The existing speaker test fixture now supplies spk_kwargs, matching real AutoModel instances. No automatic device selection, runtime migration, or model-kernel changes are included.

LauraGPT

Added a real CUDA/CPU placement check for aa353d6, beyond the registry stand-ins. On an H100 with torch/torchaudio 2.11.0+cu128, Paraformer ASR ran on cuda:0 while explicit CPU FSMN-VAD, CT-punc and CAM++ remained on CPU. Read-only forward pre-hooks on real Linear/Conv layers observed matching input/weight devices for all four roles in both calls; no models, inference methods or ComputeScores implementations were replaced.

The first call used construction placement; the second passed shared runtime device="cpu". Both retained ASR on GPU and all three submodels on CPU, returning nonempty text, word timestamps and seven speaker-labelled sentence entries. The original caller dictionaries stayed unchanged. A separate real-weight construction control using base 486b4b7 put all four models on CUDA despite the explicit CPU submodel options. That base control did not run inference; caller dictionaries also stayed unchanged in this real hub-loading path, so the earlier stand-in dictionary-mutation result should not be generalized to every path.

Reproducibility: exact candidate/base AutoModel files were Git-blob verified, overlaid on verified FunASR 1.4.15 support (443 RECORD entries); all 34 files across the existing pinned model snapshots were SHA-256 checked, with offline local paths and no weight downloads. Model revisions: Paraformer d7811ee3ac581fbcfdeb37c98c6ba674028433dc, VAD df20e6b30c653645fa4ff125cacfcabd1020a669, punctuation d0e55e2b8722a78b63705ff443d09c4f86e5d750, speaker e4b6ede7ce16997aff4ae69fbca1f0175e2afede. Input was the existing 20-second 16 kHz AISHELL-4 excerpt, SHA-256 7b3dad101a169506bc274fefa7fb7db278e941f24e780d2a55c50d23c4c23937.

Limits: this is exact AutoModel plus released support, not a complete exact-head tree run. Default frontend settings were retained; the two outputs differed (67 versus 66 timestamp pairs), so this does not establish transcript equality or accuracy and does not attribute that difference to the runtime option. This is not an MPS/M2 test, speedup/capacity benchmark, or validation of all model families. The requested original M2 workload measurement remains outstanding; the PR remains draft.

damnwenxi · 2026-09-14T14:05:41Z

Added an Apple M4 / MPS mixed-device validation using a local 71m36s audio file (4295.947s), with the PR checkout imported directly from the local PR checkout.

The validation script ran each mode in a fresh subprocess with HF_HUB_OFFLINE=1 and MODELSCOPE_OFFLINE=1. For each mode it first ran a 10s warmup clip, then two full-length runs of the 71m36s clip. The first full run passed a shared runtime device="cpu" override; the second full run used no runtime device override. Read-only encoder forward hooks checked that input tensors and encoder weights were on the expected device for ASR, VAD and punctuation.

Environment:

macOS / Apple M4
torch 2.14.0
Python 3.14.6
FunASR 1.4.15 from this PR checkout
Models: Paraformer ASR + FSMN-VAD + CT-punc
No weight downloads during validation

Results:

mode	placement	run1 (`device="cpu"` runtime override)	run2 (no override)
`all_mps`	ASR=MPS, VAD=MPS, Punc=MPS	161.87s	205.87s
`mixed_vad_cpu`	ASR=MPS, VAD=CPU, Punc=MPS	84.38s	123.43s
`mixed_both_cpu`	ASR=MPS, VAD=CPU, Punc=CPU	109.08s	166.74s

All three modes produced matching output summaries:

characters: 23115
word timestamps: 20920
sentence entries: 2111
text SHA-256: 30de7e36183fef3a638746301d16f8ecf1d54cd8dbcaa870cae7dcf1e94c2fc0

The mixed_vad_cpu configuration was fastest on this workload:

run1: 161.87s -> 84.38s, about 1.92x faster / 47.9% less time
run2: 205.87s -> 123.43s, about 1.67x faster / 40.0% less time

This confirms real MPS/CPU mixed-device inference on Apple Silicon: ASR stayed on MPS, explicit CPU VAD stayed on CPU, punctuation stayed on its configured device, and the shared runtime device="cpu" override did not overwrite the already-loaded model placements. Caller-owned submodel config dictionaries also remained unchanged.

Limitations: this is an Apple M4 local-audio validation, not the original 121-minute Apple M2 workload from #3701, and it only covers this Paraformer + FSMN-VAD + CT-punc setup. The timing numbers should be treated as workload-specific rather than a general benchmark across all model families.

LauraGPT

Rechecked eee6361. It is one commit ahead of aa353d6 with only test_submodel_device.py changed. The AutoModel Git blob is identical at both heads (61f7ce8b92116c781d6c1a6ecf517fc80ff3b46b). The four updated device tests pass against the exact candidate AutoModel module with released1.4.15 support modules. Their CUDA availability is mocked for configuration labels; all stand-in weights remain CPU. I did not repeat the already completed H100 model validation or claim an independent MPS run.

Thanks for the additional M4 report in comment5665255435 and for distinguishing it from the original M2 workaround. I have recorded the 71m36s local-audio, three-placement/two-run results as contributor-reported, workload-specific evidence, not an independently reproduced benchmark. The PR is now marked ready for review rather than draft.

The test cleanup change introduces no new blocker in this bounded check. My earlier H100 evidence retains its original limitations, including differing transcript text between the two calls under retained default frontend settings. No general accuracy, speedup or all-model acceptance is implied.

fix: honor submodel device placement across inference calls

aa353d6

LauraGPT reviewed Sep 14, 2026

View reviewed changes

test: avoid Python 3.11-only unittest cleanup API

eee6361

damnwenxi marked this pull request as ready for review September 14, 2026 14:06

LauraGPT reviewed Sep 14, 2026

View reviewed changes

Navigation Menu

Uh oh!

fix: honor submodel device placement across inference calls - #3706

fix: honor submodel device placement across inference calls#3706
damnwenxi wants to merge 2 commits into
modelscope:mainfrom
damnwenxi:fix/submodel-device-placement

damnwenxi commented Sep 14, 2026 •

edited

Loading

Uh oh!

LauraGPT left a comment

Uh oh!

damnwenxi commented Sep 14, 2026 •

edited

Loading

Uh oh!

LauraGPT left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

damnwenxi commented Sep 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Type of change

Validation

User impact

Notes for reviewers

Uh oh!

LauraGPT left a comment

Choose a reason for hiding this comment

Uh oh!

damnwenxi commented Sep 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

LauraGPT left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

damnwenxi commented Sep 14, 2026 •

edited

Loading

damnwenxi commented Sep 14, 2026 •

edited

Loading